Skip to content

feat(cli): implement runtime rule execution - #50

Merged
thecodedrift merged 12 commits into
jakob/runtime-rule-executionfrom
jakob/runtime-rule-execution-impl
Jul 6, 2026
Merged

feat(cli): implement runtime rule execution#50
thecodedrift merged 12 commits into
jakob/runtime-rule-executionfrom
jakob/runtime-rule-execution-impl

Conversation

@thecodedrift

@thecodedrift thecodedrift commented Jul 3, 2026

Copy link
Copy Markdown
Member

Implements the runtime-rule-execution OpenSpec change — the local harness that evaluates a runtime rule in taskless check.

Stacked on #49 (jakob/runtime-rule-execution, the proposal) → base this against that branch, not main.

All task groups complete (per the repo's /opsx:apply convention — commit + pause between groups):

  • 1. Runtime-rule recognitiondiscover.ts + contract types (types/runtime-rule.ts)
  • 2. Narrow → gate → check harnessnarrow.ts (one ast-grep scan/mode), invoke.ts (check.ts via tsx, timeout + throw isolation), harness.ts (FindingCheckResult)
  • 3. tsx bundling — CLI dep, runtime-resolved, externalized from the Vite bundle
  • 4. Reconcile scoping & materializationruntime/run-set.ts: sign check.ts only, per-rule eligibility, materialize blessed into .run/
  • 5. check dispatch & modescheck.ts rewrite: static always runs, runtime gated by the mode table, --dangerously-run-scripts + --timeout, --json skipped. Cutover: removed static-reconcile gating (deleted run-set.ts).
  • 6. Help & docscheck.txt (topic v2) + ci.txt
  • 7. Tests & verificationruntime-harness.test.ts + runtime-check.test.ts

Model: static ast-grep rules are trusted/inert and always run; a runtime rule's check.ts is arbitrary code and runs only when server-verified (authed reconcile) or under --dangerously-run-scripts. Only the check.ts is signed/reconciled.

typecheck + lint + full test green (338 tests). Also fixed two real bugs found while testing: a FindingCheckResult off-by-one, and a timeout that killed only the tsx wrapper (now SIGKILLs the process group).

Refs TSKL-245

Stack (root → tip):

thecodedrift and others added 5 commits July 3, 2026 08:35
Group 1 of runtime-rule-execution. Add the harness-side recognition of runtime
rules: a directory under .taskless/runtime-rules/ holding kind: runtime capture
*.yml plus a check.ts.

- src/types/runtime-rule.ts mirrors the structural harness<->check contract
  (Finding, Match, CheckFunction, CaptureRule, metadata block) from the
  generator's @taskless/types; a delivered check imports nothing, so the
  contract is structural.
- src/rules/runtime/discover.ts enumerates .taskless/runtime-rules/, parses each
  capture *.yml, confirms kind: runtime, and returns a typed RuntimeRule
  (capture rules with id/name/language/match + the check.ts path).
  .taskless/runtime-rule-tests/ is never enumerated.

Refs TSKL-245
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Groups 2-3 of runtime-rule-execution. Implement the local harness that
evaluates a runtime rule, plus the bundled tsx loader it runs check.ts under.

- narrow.ts: run a rule's capture rules as ONE ast-grep scan (anchor
  --json=stream, broad --files-with-matches) and normalize matches to the
  contract shape (0-indexed -> 1-indexed, ruleId -> model name, captures from
  metaVariables). Uses a temp --config rules dir so multiple captures + full
  ast-grep config run in a single scan.
- invoke.ts: run check.ts's default export (root, matches) via a pinned tsx
  resolved at runtime (no repo toolchain). An embedded ESM runner writes the
  returned Finding[] to an out-file; a throw, non-zero exit, or timeout is
  isolated to an error result. Default 10s bound, overridable.
- harness.ts: narrow -> gate-on-matches -> invoke -> map Finding to CheckResult
  (source: taskless-runtime); process-per-check, sequential.
- Add tsx to the CLI dependencies (externalized from the Vite bundle).

Verified end-to-end against a temp-dir fixture: discovery, narrow (correct
line normalization + rule attribution), tsx invocation, and finding mapping.

Refs TSKL-245
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d rules

Group 4 of runtime-rule-execution. Add the runtime reconcile-set: sign each
rule's check.ts (only) for reconciliation, select the rules the server blessed,
and materialize them for read-hash-execute.

- src/rules/runtime/run-set.ts: signRuntimeChecks (check.ts only — capture *.yml
  are inert), reportRuntimeChecks -> { file, signature }, selectBlessedRuntimeRules
  (content-join: a rule runs iff its check.ts signature is in the run set; the
  rest are withheld/advisory), and materializeRuntimeRules (copy blessed rule
  dirs into .taskless/.run/runtime-rules/ and re-discover so execution uses the
  blessed bytes; .run/ stays gitignored).
- discover.ts: extract discoverRuntimeRulesIn(root) so materialized rules can be
  re-discovered from .run/.
- narrow.ts: copy the original capture *.yml bytes into the temp config instead
  of re-serializing the parsed object — a YAML round-trip can alter an exotic
  ast-grep config.

Verified end-to-end: report only check.ts, bless on signature match, materialize
to .run/, execute the materialized copy; empty run set withholds the rule.

Refs TSKL-245
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ile gating

Group 5 of runtime-rule-execution. Rewire `taskless check` around the two rule
classes and complete the cutover to signing only untrusted code.

- Static ast-grep rules under .taskless/rules/ always run (trusted, no network).
- Runtime rules run only on a validated path: planRuntime resolves the mode from
  auth state + flags — authed reconcile runs blessed rules and withholds the rest
  (advisory); logged-out / --anonymous / no-remote / reconcile-unavailable skip
  runtime with a notice; --dangerously-run-scripts runs all runtime rules with no
  network behind a loud warning.
- Add --dangerously-run-scripts and --timeout <seconds>; runtime findings merge
  into the same results and exit-code logic; --json gains an additive optional
  `skipped` array (schema updated), warnings/notices stay stderr-only.
- Fix a Finding->CheckResult off-by-one: findings are 1-indexed, CheckResult.range
  is 0-indexed (display/json add 1).

Cutover: remove the stacked-under static-reconcile gating — delete
src/rules/run-set.ts and the now-obsolete test/reconcile-check.test.ts +
test/run-set.test.ts (runtime-dispatch tests land in Group 7). Static rules are
no longer signed or gated; only runtime check.ts is.

Verified end-to-end via the built CLI: static-only-runs (runtime skipped +
notice), --dangerously-run-scripts (both run), and --json (skipped array,
warnings suppressed). Full CLI suite green (326 tests).

Refs TSKL-245
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Groups 6-7 of runtime-rule-execution.

Docs (Group 6):
- check.txt (topic v2): static rules always run; runtime check.ts runs only when
  server-verified; the mode table, --dangerously-run-scripts, --timeout, and the
  --json skipped array.
- ci.txt: unauthenticated CI runs static rules and skips runtime; the
  TASKLESS_TOKEN backstop is the enforcement point for runtime check.ts.

Tests (Group 7):
- runtime-harness.test.ts: discovery, gate-on-zero-matches (check never invoked),
  match normalization + Finding->CheckResult indexing, throwing-check isolation,
  timeout -> error finding.
- runtime-check.test.ts: end-to-end dispatch via the built CLI with a mock
  reconcile server + git origin — authed-blessed, empty-run withheld, logged-out
  and --anonymous skip + report, reconcile-unavailable skips, dangerously-run-
  scripts runs offline; asserts static always runs and only check.ts is reported.

Also fix a real timeout bug found by the harness test: tsx re-execs node as a
grandchild, so spawn detached and SIGKILL the whole process group — otherwise a
runaway check keeps running past the timeout. Full suite green (338 tests).

Refs TSKL-245
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@thecodedrift
thecodedrift marked this pull request as ready for review July 3, 2026 17:53
Copilot AI review requested due to automatic review settings July 3, 2026 17:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements local runtime rule execution for taskless check, adding a harness that runs runtime rules’ check.ts only when server-verified (or explicitly forced), while continuing to always run static ast-grep rules.

Changes:

  • Add runtime rule discovery, narrowing (ast-grep), isolated check.ts invocation via bundled tsx, and FindingCheckResult mapping.
  • Rewrite taskless check dispatch to always run static rules, gate runtime rules by auth/reconcile (or --dangerously-run-scripts), and expose skipped runtime rules in --json.
  • Add unit/integration tests and update CLI help/docs to reflect the new runtime rule model.

Reviewed changes

Copilot reviewed 17 out of 18 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
pnpm-lock.yaml Locks the new tsx dependency version used to run runtime checks.
packages/cli/package.json Adds tsx as a runtime dependency for executing check.ts.
packages/cli/src/types/runtime-rule.ts Introduces the runtime rule contract types (Match, Finding, etc.).
packages/cli/src/schemas/check.ts Extends --json output schema with optional skipped runtime rules.
packages/cli/src/rules/runtime/discover.ts Discovers runtime rule directories and loads capture rule metadata.
packages/cli/src/rules/runtime/narrow.ts Runs the runtime rule “narrow” via ast-grep and normalizes matches.
packages/cli/src/rules/runtime/invoke.ts Executes check.ts under tsx with timeout + error isolation.
packages/cli/src/rules/runtime/harness.ts Orchestrates narrow→gate→invoke and maps findings to CheckResult.
packages/cli/src/rules/runtime/run-set.ts Signs/reports check.ts only and materializes blessed runtime rules into .run/.
packages/cli/src/commands/check.ts Rewrites check to always scan static rules and conditionally run runtime rules, surfacing skipped runtime rules in --json.
packages/cli/src/help/check.txt Documents runtime rules, modes, --dangerously-run-scripts, --timeout, and skipped JSON output.
packages/cli/src/help/ci.txt Updates CI guidance: static rules always run; runtime rules require auth/verification.
packages/cli/test/runtime-harness.test.ts Adds unit-ish tests for discovery/gating/mapping/timeout behavior.
packages/cli/test/runtime-check.test.ts Adds integration tests for check mode table and reconcile interactions.
packages/cli/test/run-set.test.ts Removes obsolete tests for the deleted static run-set gating logic.
packages/cli/test/reconcile-check.test.ts Removes obsolete reconciliation behavior tests superseded by runtime gating behavior.
openspec/changes/runtime-rule-execution/tasks.md Marks the OpenSpec task groups as completed and records implementation notes.
Files not reviewed (1)
  • pnpm-lock.yaml: Generated file

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread packages/cli/src/rules/runtime/run-set.ts
Comment thread packages/cli/src/rules/runtime/narrow.ts Outdated
Comment thread packages/cli/src/rules/runtime/invoke.ts
thecodedrift and others added 3 commits July 3, 2026 11:59
…ntation

Verification (opsx:verify) caught a spec-vs-impl divergence: the narrow
requirement, proposal, and design named `--inline-rules --json=stream`, but the
harness assembles the capture rules into a temp `--config` (--inline-rules
carries only one rule; a runtime rule has multiple capture rules + full ast-grep
config). Reworded to "one scan per mode" via a generated config so the spec that
gets synced to canonical on archive matches reality.

Refs TSKL-245
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sync the runtime-rule-execution delta specs into the canonical specs and archive
the change (verify → sync → archive).

- cli-runtime-rule-execution: new canonical spec (the runtime harness contract).
- cli-check: static-vs-runtime dispatch, the validated-path rule, skip+report,
  and --dangerously-run-scripts added; auth-state/reconcile/degrade requirements
  updated for the cutover; the now-obsolete 'warns on reconciliation mismatches'
  and 'exits cleanly when the run set is empty' requirements removed (their
  static-reconcile behavior was deleted in the cutover — the delta records the
  removal with reasons).
- cli-rule-reconciliation: reporting + run-set requirements rescoped to each
  runtime rule's check.ts.

Change archived to openspec/changes/archive/2026-07-03-runtime-rule-execution/;
no unarchived changes remain, so the tip's check-openspec-archived job passes.

Refs TSKL-245
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 28 changed files in this pull request and generated 4 comments.

Files not reviewed (1)
  • pnpm-lock.yaml: Generated file

Comment thread packages/cli/src/rules/runtime/discover.ts Outdated
Comment thread packages/cli/src/rules/runtime/narrow.ts Outdated
Comment thread packages/cli/src/rules/runtime/narrow.ts Outdated
Comment thread packages/cli/src/commands/check.ts Outdated
thecodedrift and others added 4 commits July 4, 2026 11:18
Six fixes from Copilot's review of #50, plus the canonical-spec twin of the #49
doc fix:

- invoke.ts: on Windows, kill the check via taskkill /T (negative PIDs aren't
  supported), so a timeout actually terminates the tsx+node tree.
- discover.ts: pin the check file to check.ts inside the rule dir; do not resolve
  metadata.taskless.check as a path (prevents escaping the dir via ../).
- run-set.ts: signRuntimeChecks is now per-rule resilient (returns unreadable
  rules instead of throwing); reported reconcile paths are POSIX-normalized so
  Windows backslashes don't defeat the server-side path match.
- check.ts: a missing/unreadable check.ts is reported as skipped and
  materialization errors degrade to a runtime-skip — a malformed runtime rule no
  longer aborts the whole check (static keeps running).
- narrow.ts: run one broad scan per broad capture rule so matches are attributed
  to the right rule (was mislabeling all broad matches as the first rule); treat
  a signal-killed narrow (null exit code) as a failure instead of success.
- specs/cli-runtime-rule-execution: fix the materialize requirement's live-tree
  path (.taskless/rules/ -> .taskless/runtime-rules/) to match #49.

Add an integration test: a runtime rule missing check.ts is skipped (not fatal)
and static rules still run. Full suite green (339).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Propagate the #49 review-comment fixes into the canonical specs (the archived
change copy came via merge):

- cli-check: degrade --json scenario allows the additive skipped array;
  --dangerously-run-scripts described as 'without server validation'.
- cli-rule-reconciliation: heading renamed to 'Reconcile reports every runtime
  rule's check.ts'.
- cli-runtime-rule-execution: per-capture match mode; broad matches are
  path-only (line/column 1, empty text/captures).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@thecodedrift
thecodedrift merged commit a1b4203 into jakob/runtime-rule-execution Jul 6, 2026
2 checks passed
@thecodedrift
thecodedrift deleted the jakob/runtime-rule-execution-impl branch July 6, 2026 20:45
thecodedrift added a commit that referenced this pull request Jul 6, 2026
Six fixes from Copilot's review of #50, plus the canonical-spec twin of the #49
doc fix:

- invoke.ts: on Windows, kill the check via taskkill /T (negative PIDs aren't
  supported), so a timeout actually terminates the tsx+node tree.
- discover.ts: pin the check file to check.ts inside the rule dir; do not resolve
  metadata.taskless.check as a path (prevents escaping the dir via ../).
- run-set.ts: signRuntimeChecks is now per-rule resilient (returns unreadable
  rules instead of throwing); reported reconcile paths are POSIX-normalized so
  Windows backslashes don't defeat the server-side path match.
- check.ts: a missing/unreadable check.ts is reported as skipped and
  materialization errors degrade to a runtime-skip — a malformed runtime rule no
  longer aborts the whole check (static keeps running).
- narrow.ts: run one broad scan per broad capture rule so matches are attributed
  to the right rule (was mislabeling all broad matches as the first rule); treat
  a signal-killed narrow (null exit code) as a failure instead of success.
- specs/cli-runtime-rule-execution: fix the materialize requirement's live-tree
  path (.taskless/rules/ -> .taskless/runtime-rules/) to match #49.

Add an integration test: a runtime rule missing check.ts is skipped (not fatal)
and static rules still run. Full suite green (339).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot mentioned this pull request Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants